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ABSTRACT:  A  U.S.  Navy  enterprise  methodology  has  been  established  for  consistent  ship  self  defense  operational  test 
and  evaluation  (OT&E)  across  all  ship  classes.  A  cornerstone  of  the  Navy  T&E  Enterjrrise  is  the  Probability  of  Raid 
Annihilation  (Pra)  Testbed.  The  Navy  PRA  Testbed  implements  HLA  federated  simulations  of  ship  combat  system  ele¬ 
ments  against  independent,  reactive  threat  raids  in  a  common  environment  to  formulate  an  overall  combat  system  as¬ 
sessment.  The  LPD  17  ship  class  is  the  first  to  implement  a  Pra  Testbed  baseline  as  a  formal  component  of  ship  class 
OT&E.  Products  and  lessons  from  the  LPD  17  baseline  are  being  transitioned  to  multiple  ship  classes  including 
DDG  1000,  LHA  6,  LCS,  and  CVN  21. 

While  the  Pra  Testbed  LPD  17  baseline  adheres  to  the  Federation  Development  and  Execution  Process  (FEDEP),  con¬ 
ceptual  modeling  is  a  particularly  distinctive  feature.  The  Systems  Engineering  Concept  Model  (SECM)  is  an  extension 
of  the  ‘ Conceptual  Model’  product  of  FEDEP  step  2  that  broadens  the  impact  of  conceptual  modeling  in  the  SE  process 
and  specifically  supports  federation  Verification,  Validation,  and  Accreditation  (W&A).  Accreditation  of  the  PRA  Test¬ 
bed  for  LPD  17  operational  testing  is  a  challenging  proposition.  The  purpose  of  the  SECM  is  to  provide  a  central 
documentation  tool  for  stakeholders  to  ‘view’  the  Testbed  capabilities  and  limitations  from  multiple  perspectives. 

The  SECM  is  divided  into  3  views,  which  generally  follow  sequential  stages  in  Testbed  system  engineering:  System 
View,  Model  View,  and  Federate  View.  Each  view  is  intended  to  reveal,  address,  and  resolve  a  particular  class  of  issues 
for  Pra  Testbed  accreditation.  Explanation  of  engineering  judgments  is  inclusive  to  the  documentation.  The  SECM  is  a 
not  an  ‘easy  button’  for  making  engineering  decisions  or  wishing  away  limitations.  Yet,  it  does  create  a  framework  of 
deliberate  intent  for  making  decisions  and  accountability  for  limits  in  capability.  The  SECM  also  establishes  a  persis¬ 
tent  documentation  product  that  feeds  the  corporate  knowledge  base  for  subsequent  ship  classes. 

During  SECM  development  many  lessons  have  been  learned  about  conceptual  modeling.  In  this  paper,  we  venture  to 
share  those  lessons  learned  and  describe  how  we  overcame  the  trials  and  tribulations  of  documenting  engineering  level 
details  in  the  SECM  for  the  PRA  Testbed  LPD  1 7  baseline. 

1.  Introduction 

The  U.S.  Navy  has  moved  to  more  of  an  ‘enterprise’  ap¬ 
proach  to  conduct  developmental  and  operational  testing 
for  Ship  Self  Defense.  This  effort  endeavors  to  eliminate 
or  decrease  duplication  in  both  testing  and  model  devel¬ 


opment.  It  also  endeavors  to  increase  re-use  of  tools  and 
simulation  frameworks  to  decrease  developmental  and 
operational  testing  cost  for  the  Navy.  The  enterprise  ap¬ 
proach  to  T&E  has  been  formalized  in  a  Capstone  Enter¬ 
prise  Air  Warfare  Ship  Self  Defense  (AW  SSD)  Enter¬ 
prise  Test  and  Evaluation  Master  Plan  (TEMP)1.  The  SSD 
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Enterprise  TEMP  was  drafted  during  mid-FY06  and  has 
been  signed  by  the  Program  Executive  Officer,  Integrated 
Warfare  Systems  (PEO IWS)  and  the  Navy’s  Com¬ 
mander,  Operational  Test  and  Evaluation  Force  (COTF). 

2.  Ship  Self  Defense  T&E  Enterprise 

The  Ship  Class  Anti-Air  Warfare  (AAW)  Self-Defense 
Capstone  Requirements  Document  (CRD)  of  5  Feb  1996 
defines  a  Probability  of  Raid  Annihilation  (Pra)  require¬ 
ment  against  Anti-Ship  Cruise  Missiles  (ASCM)  for  sur¬ 
face  ships2.  The  requirement  applies  to  several  new  ship 
classes,  including  LPD  17,  LHA  6,  DDG  1000,  CVN  21, 
and  the  Littoral  Combat  Ship  (LCS). 

Meeting  Pra  requirements  involves  a  significant  amount 
of  ship  and  combat  systems  level  T&E.  In  addition,  indi¬ 
vidual  Combat  System  Element  Program  Offices  have 
specific  AW  SSD  T&E  requirements  to  evaluate  proper 
combat  functionality  and  performance  against  Anti-ship 
Cruise  Missile  (ASCM)  threats2.  Integrated  ship  self  de¬ 
fense  performance  assessment  poses  certain  distinct  chal¬ 
lenges  to  the  T&E  community.  Among  them  are  short 
range  hardkill  engagements  and  integrated  hard- 
kill/softkill  engagements,  both  against  multi-threat  ASCM 
presentations.  The  confluence  of  issues  results  in  no  live 
test  venue  being  able  to  fully  address  end-to-end  perform¬ 
ance  of  the  complete  combat  system.  Hence,  M&S  is 
needed  to  augment  live  results  and  formulate  the  overall 
combat  system  PRA  assessment. 

The  Enterprise  TEMP  consolidates  testing  across  ship 
classes  according  to  combat  system  variants  vice  plat¬ 
forms  and  implements  M&S  in  Navy  surface  ship  opera¬ 
tional  testing  at  an  unprecedented  level.  The  Enterprise 
TEMP  pursues  culture  change  in  Navy  T&E  by  indoctri¬ 
nating  Pra  Testbed  simulation  events  as  formal  test  events 
in  the  Enterprise  TEMP.  Further,  the  Enterprise  TEMP 
coordinates  the  M&S  events  with  a  carefully  planned  se¬ 
ries  of  empirical  test  events  across  the  combat  system 
variants  and  ship  classes.  Centralized  planning  ensures 
adequate  live  testing  to  validate  the  models  and  assess 
performance  without  redundant  testing. 

2.1  Enterprise  PRA  Testbed 

A  cornerstone  of  the  Navy  T&E  Enterprise  is  the  Pra 
Testbed.  The  Navy  PRA  Testbed  implements  High  Level 
Architecture  (HLA)  federated  simulations  of  ship  combat 
system  elements  against  independent,  reactive  threat  raids 
in  a  common  environment  to  formulate  an  overall  combat 
system  performance  assessment2. 

PEO  IWS  is  responsible  for  ensuring  consistency  and 
continuity  of  the  assessment  process  and  its  associated 
simulation  framework.  The  simulation  framework  that 


guides  Pra  Testbed  development  is  governed  by  the  fol¬ 
lowing  top-level  requirements: 

1 .  Interoperable  simulations  on  a  single  runtime  in¬ 
frastructure 

2.  All  system  representations  execute  simultane¬ 
ously  for  each  ship  defense  engagement 

3.  Common  threat  and  natural  environment 
achieved  via  unified  modeling,  distributed  exe¬ 
cution 

4.  System-to-system  communications  should  be  In¬ 
terface  Design  Specification  (IDS)  compliant 

5.  System-to-system  interactions  (e.g.,  signal 
propagations,  emissions  detections)  should  be 
physics-based  through  the  common  environment 

6.  Representation  detail  must  be  of  sufficient  detail 
for  use  in  Operational  Test  and  Evaluation 
(OT&E). 

The  functional  components  of  a  PRA  Testbed  baseline  are 
depicted  in  Figure  1.  The  Pra  Testbed  architecture  pro¬ 
vides  simulation  runtime  mechanisms  for  interfacing 
combat  system  element  simulations,  implementing  com¬ 
mon  threat  and  environment  representations,  and  realizing 
integrated  hardk  i  I  l/softk  i  1 1  scenarios.  Each  component 
must  provide  representation  of  sufficient  detail  to  achieve 
accreditation  for  use  in  testing.  Specific  simulation  re¬ 
quirements  for  particular  components  are  resolved  during 
the  Testbed  baseline  system  engineering  process. 
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Figure  1-  Pra  Testbed  Components 

Pra  is  assessed  by  executing  the  virtual  ship/platform 
within  the  virtual  range  across  an  array  of  scenarios  and 
threat  types.  Each  execution  of  the  Pra  Testbed  yields  the 
result  for  one  threat  raid  engagement.  A  matrix  of  runs 
across  the  span  of  scenarios  and  threat  types  yields  the 


overall  PRA  score.  The  simulations  are  validated  with  live 
test  events  including:  land-based  testing,  laboratory  tests, 
element-level  tests,  at-sea  testing  with  the  Operational 
Ship,  and  at-sea  testing  with  the  Self  Defense  Test  Ship 
(SDTS). 

A  baseline  of  the  PRA  Testbed  will  be  implemented  for 
each  unique  new  ship  class  combat  system  configuration. 
Specific  test  planning  for  each  PRA  Testbed  baseline  im¬ 
plementation  is  described  in  Section  III  of  the  SSD  Enter¬ 
prise  TEMP2.  The  LPD  17  ship  class  is  the  first  to  imple¬ 
ment  a  PRA  Testbed  baseline  as  a  formal  component  of 
ship  class  operational  test  &  evaluation.  Simulation  prod¬ 
ucts  and  lessons  from  the  LPD  17  baseline  are  being  tran¬ 
sitioned  to  follow-on  ship  classes,  dramatically  reducing 
risk  and  cost  of  ship  class  operational  evaluation. 

3.  The  Systems  Engineering  Concept  Model 

PRA  Testbed  systems  engineering  follows  the  IEEE  1516 
standard  Federation  Development  and  Execution  Process 
(FEDEP)  for  High  Level  Architecture  (HLA)  simulation. 
The  Systems  Engineering  Concept  Model  (SECM)  is  an 
extension  of  the  ‘Conceptual  Model’  product  of  the  early 
stages  of  the  Testbed  system  engineering  process.  The 
SECM  establishes  a  persistent  documentation  product  that 
broadens  the  impact  of  conceptual  modeling  further  in  the 
SE  process  and  specifically  supports  federation  Verifica¬ 
tion,  Validation,  and  Accreditation  (VV&A).  The  SECM 
also  provides  a  mechanism  to  examine  shared  modeling 
responsibilities  (e.g.,  threat,  natural  environment)  and 
understand  migration  of  ‘stand  alone’  legacy  models  to 
the  PRA  Testbed  federation. 

A  number  of  factors  combined  to  incite  development  of 
the  SECM  as  a  documentation  tool  to  support  accredita¬ 
tion  of  the  PRA  Testbed: 

•  The  ship  self  defense  problem  space  is  complex 

•  Systems  performance  for  PRA  assessment  span 
across  different  technical  communities  and  multi¬ 
ple  managing  program  offices 


•  PRA  will  be  assessed  using  a  federation  of  interop¬ 
erable  simulations;  it  will  not  (cannot)  be  tested 
empirically 

•  Many  specific  parameters,  assumptions,  limita¬ 
tions,  etc.  are  negotiated  between  the  testing  and 
acquisition  communities 

•  The  testing  community  is  intent  on  consistent  PRA 
assessment  across  ship  classes  and  combat  system 
configurations. 

Accreditation  of  the  PRA  Testbed  for  use  in  Operational 
Test  and  Evaluation  is  a  challenging  proposition.  The 
purpose  of  the  SECM  is  to  provide  a  central  documenta¬ 
tion  tool  for  stakeholders  to  ‘view’  the  Testbed  capabili¬ 
ties  and  limitations  from  multiple  perspectives. 
Stakeholders  include:  COTF,  the  DoD’s  Director  of  Test 
&  Evaluation  (DOT&E),  PEO  IWS,  ship  class  Program 
Offices,  combat  system  element  Project  Offices,  subject 
matter  experts,  and  the  Testbed  engineers  themselves. 

The  SECM  provides  a  single,  navigable  documentation 
product  utilizing  visual  and  narrative  descriptions  of  both 
‘what’  and  ‘why’  for  the  PRA  Testbed.  Explanations  of 
engineering  judgments  are  inclusive  to  the  documenta¬ 
tion. 

4.  The  Three  Views  of  the  SECM 

The  SECM  is  divided  into  3  views,  which  generally  fol¬ 
low  sequential  stages  in  Testbed  system  engineering: 

•  System  View 

•  Model  View 

•  Federate  View. 

Each  view  of  the  SECM  is  intended  to  reveal,  address, 
and  resolve  a  particular  class  of  issues  for  PRA  Testbed 
accreditation  (see  Figure  2).  By  keeping  issues  cleanly 
divided  and  addressing  each  in  depth  the  SECM  provides 
an  able  documentation  tool  to  support  PRA  Testbed  system 
engineering  and  accreditation  for  use  in  OT&E. 
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Figure  2.  Systems  Engineering  Concept  Model  Views 


4.1  System  View 

The  System  View  of  the  SECM  describes  the  real  world 
systems  and  expresses  the  limit  of  understanding  of  the 
real  world  systems  to  be  modeled.  In  practical  terms,  the 
System  View  provides  the  first  insight  into  the  scope  of 
the  problem  space  to  be  modeled.  The  System  view  ex¬ 
plains  what  systems  and  platforms  are  to  be  included  in 
the  PRA  assessment,  which  aren’t,  and  why.  The  System 
view  explains  the  scenarios  to  be  applied  and  what  as¬ 
sumptions  are  included.  This  view  provides  a  basis  upon 
which  to  choose  models  to  represent  the  required  systems. 

Example:  The  PRA  metric  is  expressed  as  a  requirement 
for  single-ship  self  defense  capability  against  a  given 
threat  exposure.  Definition  of  environmental  conditions, 
indication  and  warning,  and  ownship/threat  tactics  are 
negotiated  among  the  testers  and  ship  class  representa¬ 
tives.  Results  of  these  negotiations,  of  course,  can  have 
tremendous  implications  on  simulation  design  decisions 
later  in  the  system  engineering  process.  The  SECM  cap¬ 
tures  the  results  of  these  negotiations,  along  with  their 
rationale,  in  the  System  View. 

Example:  The  LPD  17  sensor  suite  includes  two  primary 
active  radars  (SPQ-9B  surface  search  radar,  SPS-48E  vol¬ 
ume  search  radar),  an  electronic  warfare  receiver  (SLQ- 
32),  and  other  systems  such  as  the  SPS-73  navigational 
radar.  For  LPD  17  PRA  assessment  the  testers  and  ship 
class  representatives  agreed  to  include  only  the  SPQ-9B, 
SPS-48E,  and  SLQ-32  systems  in  the  LPD  17  perform¬ 
ance  assessment.  Other  systems  such  as  the  SPS-73  were 
disregarded.  This  decision,  along  with  its  rationale,  is 
captured  in  the  SECM  System  View.  Further,  resulting 
constraints  on  various  system  representation  requirements 
are  captured  in  this  view,  e.g.,  the  sensor  fusion  system 
representation  will  not  be  required  to  include  an  SPS-73 
interface. 

The  System  View  section  of  the  SECM  captures  a  number 
of  results  from  negotiation  between  the  testers  and  ship 
class  representatives  to  establish  boundaries  of  the  PRA 
assessment.  In  this  regard,  the  System  View  informs,  and 
is  informed  by  the  early  objectives  and  requirements  defi¬ 
nition  activities  of  FEDEP  Step  1.  As  such,  the  SECM 


pushes  conceptual  modeling  earlier  in  the  FEDEP  than 
formally  called  for  in  Step  2. 

4.2  Model  View 

The  Model  View  of  the  SECM  describes  the  models  to  be 
used  in  the  PRA  Testbed  in  mathematical  and  algorithmic 
terms.  It  describes  equations,  algorithms,  and  procedures 
to  provide  a  basis  for  understanding  the  quality  of  system 
representations  to  be  used  in  the  Testbed.  It  provides  in¬ 
sights  into  the  strengths  and  limitations  of  the  mathemati¬ 
cal  representations  of  the  systems. 

The  Model  View  is  concerned  with  the  capabilities  and 
limitations  of  the  models  to  be  used,  but  not  their  software 
realizations.  The  Model  View  is  not  concerned  with  func¬ 
tional  allocation  within  the  federation  or  eventual  soft¬ 
ware  implementation  in  federates.  For  example,  the 
Model  View  will  include  descriptions  of  radar  antenna 
models,  signal  propagation  calculations,  and  target  radar 
cross  section  models.  Yet,  explanation  of  whether  each  of 
these  models  is  implemented  in  different  federates  or  all 
in  a  single  federate  is  left  to  the  Federate  View  of  the 
SECM.  This  can  be  particularly  tricky,  but  also  of  utmost 
importance  when  addressing  shared  responsibilities 
(things  multiple  systems  “touch”)  such  as  representation 
of  threats  and  the  natural  environment.  For  the  PRA  Test¬ 
bed,  the  SECM  treats  the  natural  environment  as  a  dis¬ 
tinct,  independent  system  to  be  modeled.  This  puts  addi¬ 
tional  focus  on  the  requirement  to  model  the  natural  envi¬ 
ronment  consistently  across  all  systems  in  the  PRA  prob¬ 
lem  space. 

Example:  The  SLQ-32  electronic  warfare  system  in¬ 
cludes  an  operator  in  the  loop.  The  PRA  Testbed  LPD  17 
baseline  executes  much  slower  than  real-time,  so  for 
LPD  17  performance  assessment  the  operator  will  not  be 
a  live  individual.  Rather,  a  mathematical  model  will  be 
implemented  that  incorporates  varying  time  delays  to  per¬ 
form  actions.  The  SECM  Model  View  describes  the 
mathematics  used  to  represent  the  operator,  explains  the 
inherent  limitations,  and  asserts  a  case  for  adequacy  of  the 
operator  model.  It  discusses  whether  factors  such  as  train¬ 
ing  levels,  operational  stress,  and  probability  of  correct 
decisions  have  been  incorporated,  and  if  so,  how.  The 


Model  View  discussion  of  the  SLQ-32  operator  does  not 
describe  what  software  federate  includes  the  operator 
model;  that  is  for  the  Federate  View. 

Example:  The  SLQ-32  electronic  warfare  system  in¬ 
cludes  a  radio  frequency  receiver  for  detecting  threat 
emissions.  The  Model  View  description  of  the  SLQ  32 
system  representation  includes  a  discussion  of  the  SLQ- 
32  receiver  subsystem  model.  The  discussion  includes  the 
mathematical  implementation  of  the  receiver  antenna  and 
algorithms  for  signal  detection.  However,  the  Model 
View  discussion  of  the  SLQ-32  does  not  include  a  de¬ 
scription  of  how  threat  emissions  are  modeled,  though  the 
same  developer  organization  may  be  designing  those 
equations  and  those  calculations  may  ultimately  be  allo¬ 
cated  with  the  SLQ-32  model  into  the  same  federate. 
Rather,  the  threat  emission  discussions  are  segregated  to 
the  threat  modeling  section  of  the  SECM  Model  View. 
This  isn’t  done  to  make  life  hard  on  the  EW  system  mod¬ 
eler,  but  to  enforce  the  discipline  required  to  ensure  the 
threat  is  represented  consistently  across  the  entire  PRA 
Testbed. 

The  Model  View  has  been  in  many  ways  the  most  impor¬ 
tant  yet  most  troublesome  component  of  the  SECM.  Most 
simulation  developers  intrinsically  jump  from  systems 
perspective  to  software  perspective  without  much,  if  any, 
discussion  of  pure  modeling.  This  has  been  particularly 
true  when  migrating  legacy  simulations  to  the  PRA  Test¬ 
bed.  Requiring  developers  to  think,  and  document,  in  fed¬ 
eration-level  terms  segregated  from  their  “home  base” 
code  has  been  an  extremely  challenging  endeavor.  Here 
the  spiral  nature  of  the  FEDEP  has  been  beneficial,  as  it 
permits  multiple  passes  at  the  early  system  engineering 
steps  as  the  spirals  are  executed. 

4.3  Federate  View 

The  Federate  View  of  the  SECM  describes  how  the  mod¬ 
els  have  been  broken  up  and  clustered  in  the  federation 
software  (i.e.,  what  went  into  what  federates).  This  view 
provides  insights  into  the  strengths/limitations  of  the 
software  realization  of  the  models  in  the  federation.  This 
view  distinctively  reveals  the  implications  of  functional 
allocation  within  the  Testbed.  If,  for  example,  a  sensor 
model  and  target  model  are  allocated  to  separate  feder¬ 
ates,  the  implications  of  requiring  them  to  interact  via  the 
RTI  are  expressed  in  the  Federate  View.  These  include 
update  rates,  coasting  or  dead  reckoning  on  data,  and  lim¬ 
its  on  data  resolution.  The  Federate  View  also  provides 
information  on  the  Federation  Agreements,  Federation 
Object  Model,  and  coordinate  systems  to  be  used  in  the 
Testbed  federation. 

Example:  A  primary  self  defense  weapon  on  the  LPD  17 
is  the  Rolling  Airframe  Missile  (RAM).  During  PRA  Test¬ 


bed  development,  the  threat  airframe  and  RAM  models 
were  allocated  to  separate  federates.  Threat  position  and 
orientation  updates  are  published  to  the  RAM  model 
across  the  RTI.  Implications  of  this  functional  allocation 
decision  on  the  RAM  model  are  described  in  the  SECM 
Federate  View.  An  explanation  is  given  for  the  minimum 
data  rate  to  support  model  validity,  as  well  as  selection 
the  dead  reckoning  algorithm  chosen  to  compensate  for 
target  position  data  gaps. 

The  Federate  View  section  of  the  SECM  captures  impacts 
of  engineering  judgments  applied  during  functional  allo¬ 
cation.  In  this  regard,  the  Federate  View  informs,  and  is 
informed  by  the  engineering  activities  of  FEDEPS  Steps 
3-4.  As  such,  the  SECM  continues  explicit  conceptual 
model  development  later  in  the  FEDEP  than  formally 
called  for  in  Step  2. 

5.  The  SECM  and  the  Pra  Testbed  Systems 
Engineering  Process 

As  previously  stated.  The  SECM  is  an  extension  of  the 
‘Conceptual  Model'  Product  from  Step  2  (Perform  Con¬ 
ceptual  Analysis)  of  the  FEDEP.  The  SECM  concept  is  an 
effort  to  broaden  the  conceptual  model  bounds  in  the 
FEDEP  systems  engineering  process.  The  FEDEP  is 
based  on  a  spiral  development  process;  the  SECM  ven¬ 
tures  to  capture  the  spiral  maturation  of  a  simulation  from 
the  objective  and  requirements  statements  through  the 
functional  allocation  of  federates. 

Documentation  for  the  SECM  starts  with  the  Step  1  of  the 
FEDEP  (Define  Federation  Objectives).  In  this  step,  the 
objective  of  the  federation  is  stated  which  contains  the 
overall  plan  for  the  simulation.  It  defines  the  purpose  for 
the  simulation,  the  system  components  necessary  to  con¬ 
duct  the  simulation  and  what  constraints  are  required  from 
the  Threats  and  Environment  on  the  simulation. 

For  Step  2  of  the  FEDEP  (Perform  Conceptual  Analysis), 
the  SECM  links  the  outputs  from  Step  1  with  driven- 
down  information  needed  for  the  Conceptual  Model.  In 
the  Conceptual  Model,  the  requirements  listed  in  the  Fed¬ 
eration  Objective  are  reviewed  to  create  simulation  sce¬ 
narios.  These  scenarios  must  create  the  right  test  envi¬ 
ronment  to  answer  the  Federation  Objectives.  With  sce¬ 
narios  decided  upon,  the  next  part  of  Step  2  is  detailing 
the  needs  of  each  system  component  in  the  simulation.  In 
the  real-world  systems,  very  complex  physics,  system 
interactions,  and  reactive  behaviors  are  intrinsic  to  per¬ 
formance.  To  accurately  depict  these  systems  in  the  simu¬ 
lation  world,  the  SECM  endeavors  to  connect  the  real- 
world  systems  with  the  mathematical  models  that  repre¬ 
sent  them.  The  requirements  of  the  simulation  are  based 
on  the  real-world  systems,  which  then  translate  simulation 


requirements  for  the  models.  Here  the  Model  View  of  the 
SECM  is  paramount,  and  the  requirement  for  disciplined 
system  engineering  is  most  stringent. 

Once  the  Conceptual  Model  is  drafted  including  the  Fed¬ 
eration  Scenarios  and  the  Federation  Requirements,  the 
SECM  then  connects  the  functions  associated  in  the  Con¬ 
ceptual  Model  with  Step  3  of  the  FEDEP  (Design  Federa¬ 
tion).  To  properly  design  the  federation,  the  real-world 
systems  turned  mathematical  models  need  to  be  accu¬ 
rately  represented  by  their  functions.  Then  the  design  and 
functional  allocation  can  be  drafted  as  a  solution  to  an¬ 
swer  the  simulation  objective. 

With  a  drafted  Federation  Design  in  place.  Step  4  of  the 
FEDEP  (Develop  Federation)  can  begin.  Because  the 
FEDEP  is  a  spiral  development,  the  development  of  the 
federation  will  mature  from  1...N  builds  resulting  in  inte¬ 
gration  and  tests  to  properly  assess  the  validity  of  the 
simulation.  In  this  step,  the  SECM  documents  the  Federa¬ 
tion  Object  Model  (FOM),  which  details  the  HLA  objects 
used  to  represent  entities  and  interactions;  Federation 
Agreements,  which  details  the  rules  of  the  federation  and 
detailed  Federation  Scenarios  including  simulation  stan¬ 
dards. 

The  SECM  is  treated  as  a  persistent  document  throughout 
Pra  Testbed  system  engineering.  It  captures  at  each  step 
the  important  engineering  decisions  and  forces  the  engi¬ 
neering  documentation  to  not  only  describe  “what”  is 
modeled  but  also  “why”  it  is  good  enough  for  OT&E.  The 
intent  of  the  SECM  documentation  process  is  to  build  into 
the  system  engineering  the  right  artifacts  to  inherently 
support  PRA  Testbed  VV&A. 

6.  Lessons  Learned 

The  original  impetus  for  the  PRA  Testbed  SECM  was  sim¬ 
ply  to  avoid  glossing  over  the  conceptual  modeling  task 
of  early  stage  federation  system  engineering.  It  was  un¬ 
derstood  that  a  top  level  “viewgraph”  concept  model 
would  be  insufficient  rigor  for  the  federation’s  intended 
use  in  OT&E.  It  was  also  understood  that  rigor  applied 
early  in  the  system  engineering  process  would  pay  divi¬ 
dends  in  the  VV&A  process,  though  specific  mechanics 
were  not  originally  known.  However,  as  the  SECM  de¬ 
velopment  has  progressed  it  has  gained  more  persistent 
are  far-reaching  value.  It  has  been  used  to: 

•  Document  the  “story”  of  the  federation  over  a 
multi-year  project  with  changing  personnel 

•  Reveal  more  readily  the  implications  of  engineer¬ 
ing  judgments  and  programmatic  decisions  on  the 
federation 


•  Provide  a  rubric  for  maturing  federation  capability 
across  multiple  complete  federation  development 
spirals. 

•  Provide  the  single,  cohesive  body  of  engineering 
documentation  evidence  for  the  VV&A  case 

•  Establish  a  corporate  knowledge  base  for  sub¬ 
sequent  ship  classes  in  the  T&E  Enterprise. 

Along  the  way  a  number  of  lessons  have  been  learned, 
among  them: 

The  SECM  framework  format  is  key.  The  division  of 
system-model-federate  views  seems  to  make  sense  to  a 
variety  of  constituents,  including  software  developers, 
subject  matter  experts,  military  decision  makers,  and  T&E 
professionals.  The  SECM  documentation  itself  is  built 
upon  collected  components  that  are  organized  in  an 
HTML-navigable  fashion.  The  ability  to  navigate  non- 
linearly  through  information  provides  opportunities  for 
different  narrative  threads  for  different  readers  and  differ¬ 
ent  purposes.  This  approach  also  tries  to  ease  the  burden 
on  authors  to  produce  large  formal  reports,  leveraging  bits 
of  documentation  wherever  and  whenever  available.  An 
unexpected  benefit  of  this  approach  is  it’s  well-aligned  to 
spiral  documentation,  and  readily  absorbs  maturing 
documentation  alongside  maturing  federation  capability. 
Though  constituents  are  not  always  as  forthcoming  with 
specific  documents  as  liked,  they  do  understand  and  ap¬ 
preciate  the  3-view  structure  and  the  associated  documen¬ 
tation  framework. 

Decision  makers  both  love  and  hate  seeing  conse¬ 
quences  of  decisions  they  make.  The  SECM  is  remorse¬ 
less  documentation,  and  it  serves  well  as  the  conscience 
of  the  federation.  This  has  been  very  important  (particu¬ 
larly  in  the  System  View)  for  capturing  results  of  com¬ 
promises  made  by  the  testers  or  the  ship  class  program 
office  during  development.  It  provides  everyone  a  “re¬ 
ceipt”  that  prevents  re-treading  technical  ground  already 
trod  or  re-hashing  old  decisions.  This  improves  efficiency 
and  provides  a  sense  of  security  that  results  of  decisions 
are  firm  and  flow  directly  into  federation  development. 
However,  negative  consequences  on  federation  capability 
are  not  obfuscated,  which  at  times  is  irritating  to  the  deci¬ 
sion  makers. 

Engineers  tend  to  think,  and  document,  from  a  feder¬ 
ate-centric  vantage  point.  Participants  in  the  federation 
development  are  most  comfortable  discussing  their  soft¬ 
ware.  This  federate-centric  vantage  point  provides  a  home 
base  for  them,  and  it  is  an  ongoing  chore  to  segregate 
functions  from  federates.  This  is  particularly  tricky  when 
wrestling  with  functions  allocated  from  a  single  model 
into  multiple  federates.  Convincing  the  engineers  to  em¬ 
brace  the  premise  of  shared  modeling  responsibilities 
where  the  complete  realization  is  allocated  among  multi- 


pie  federates  (e.g.,  threat  representation)  continues  to  be 
vexing. 

Getting  people  to  document  “what  it  is”  is  very  hard; 
getting  them  to  document  “why  it’s  good  enough”  is 
even  harder.  Worse,  many  of  the  key  pieces  of  validation 
evidence  reside  in  the  brains  of  people  from  whom  ob¬ 
taining  thousands  of  lines  of  code  is  easier  than  tens  of 
pages  of  report.  In  this  regard,  federation  system  engi¬ 
neering  faces  the  same  documentation  problem  as  many 
technical  projects.  Yet,  the  requirement  for  federation 
accreditation  heightens  the  demand  for  descriptions  not 
just  of  “what”  by  “why”  (e.g.,  not  just  how  a  radar  an¬ 
tenna  is  modeled  but  why  that’s  good  enough  for  OT&E 
use).  The  necessity  for  written  advocacy  of  a  model  (we 
have  sometimes  referred  to  this  as  “presenting  your  case 
to  the  jury”)  is  fundamental  to  the  V&V  documentation 
package.  This  necessity  has  been  slow  to  take  hold  with 
the  PRA  Testbed  developers.  Fortunately,  the  Testbed  spi¬ 
ral  development  process  has  provided  multiple  opportuni¬ 
ties  to  redress  this  issue. 

The  SECM  has  been  very  effective  at  revealing  inter¬ 
system  gaps  and  assumptions.  Here  simulation  interop¬ 
erability  is  analogous  to  systems  interoperability.  The 
SECM  takes  a  system-of-systems  vantage  point,  and  in¬ 
sists  on  enough  detail  to  explain  each  element  system  and 
system  interaction.  Element  system  developers  typically 
live  in  the  world  of  their  own  system;  drawing  them  out  is 
challenging  but  necessary.  The  SECM  was  a  very  effec¬ 
tive  facilitator  for  creating  that  needed  cross-talk.  A  factor 
contributing  to  its  effectiveness  was  the  persistence  of  the 
SECM  throughout  the  development  process.  It  wasn’t 
perceived  by  the  engineers  as  a  “one  and  done”  item  of 
the  SE  process,  but  rather  a  key  tool  being  maintained  by 
the  Federation  System  Engineer. 

Much  information  exchange,  especially  at  the  working 
level,  is  not  inherently  documented.  Yet,  these  decisions 
can  have  profound  implications  on  federation  perform¬ 
ance  and  acceptability.  This  is  particularly  true  when  pairs 
or  groups  of  engineers  work  out  a  compromise  on  a  diffi¬ 
cult  problem.  Often  several  assumptions  get  buried  in  the 
engineering  negotiation,  and  don’t  show  up  in  any  formal 
documentation  until  goaded  out. 

A  picture  is  worth  a  thousand  spreadsheets.  This  is 
particularly  true  when  a  mix  of  constituents  is  involved; 
what  makes  intuitive  sense  to  one  group  can  be  com¬ 
pletely  perplexing  to  another.  Interoperable  simulation 
inevitably  causes  federate  developers  to  give  up  control  of 
certain  data  or  functions  they  are  accustomed  to  control¬ 
ling  when  running  alone  (e.g.,  the  radar  modelers  were 
not  accustomed  to  subscribing  to  reactive  threat  position 
data  from  an  outside  simulation).  As  obvious  as  it  seems, 
the  value  of  pictures  over  words  or  spreadsheets  can’t  be 


overstated  for  facilitating  effective  communication. 
Graphical  representation  of  FOM  data  captured  in  the 
SECM  was  specifically  helpful. 

The  SECM  done  right  automatically  collects  the  core 
V&V  documentation.  Though  it  may  be  described  as  an 
overlay,  VV&A  is  nevertheless  intrinsic  to  the  system 
engineering  process.  Accreditation  for  OT&E  is  a  high 
hurdle  to  cross;  it  involves  tough  questioning  from  T&E 
professionals  and  requires  significant  evidence  to  prove 
worthiness.  Yet,  development  of  the  SECM  includes  ask¬ 
ing  and  answering  those  same  tough  questions.  Further, 
the  Pra  Testbed  system  engineering  process  included  the 
T&E  professionals  as  direct  participants,  such  that  any 
concerns  or  issues  are  echoed  in  the  process  artifacts. 
During  LPD  17  PRA  Testbed  development,  the  SECM  - 
by  answering  questions  and  addressing  concerns  -  be¬ 
came  the  natural  repository  for  much  of  the  engineering 
documentation  needed  to  support  the  accreditation  case. 
The  real  payoff  for  the  Navy  is  the  forward  transition  of 
this  evidence  to  the  other  PRA  Testbed  baselines  in  the 
T&E  Enterprise:  consistent  system  engineering  documen¬ 
tation  across  ship  classes  used  to  maintain  hard-earned 
confidence  and  minimize  non-recurrent  engineering  costs. 

7.  Way  Forward 

The  PRA  Testbed  is  a  critical  component  of  the  U.S. 
Navy’s  Ship  Self  Defense  T&E  Enterprise;  the  Systems 
Engineering  Concept  Model  is  a  critical  component  of 
Pra  Testbed  systems  engineering.  The  SECM  pushes  con¬ 
ceptual  modeling  both  earlier  and  later  than  typically  seen 
in  the  HLA  FEDEP.  It  establishes  a  standing  documenta¬ 
tion  tool  that  is  well-suited  to  spiral  development.  The 
SECM  provides  a  single,  navigable  documentation  prod¬ 
uct  utilizing  visual  and  narrative  descriptions  of  both 
‘what’  and  ‘why’  for  the  PRA  Testbed.  Explanation  of 
engineering  judgments  is  inclusive  to  the  documentation. 
As  such,  the  SECM  has  become  a  key  enabler  of  VV&A 
for  the  PRA  Testbed. 

The  SECM  consists  of  both  a  process  and  a  product.  The 
process  is  the  same  for  all  applications  but  the  product  is 
specific  to  a  particular  application.  The  SECM  itself  cap¬ 
tures  the  information  in  a  series  of  diagrams  and  docu¬ 
ments  that  exemplify  various  aspects  of  the  particular 
application.  Although  the  SECM  artifact  itself  is  specific 
to  a  given  application,  such  as  a  PRA  Federation  Testbed, 
much  of  the  information  captured  will  be  reused  in  related 
applications.  Such  re-use  will  provide  significant  risk 
mitigation  by  promoting  standardization  and  reducing 
non-recurrent  engineering  costs. 
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